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Dabin and Taveggia (1966) described an impressive variety of 
research in wnich numerous teaching methods have shown virtually 
no reliable differences in the performance of students. Indeed, 
if it were not for the finding that study time did contribute to 
improved grades for college students, their report would have been 
even more devastating. 

Others (e.g. Markle, 196?) have reported dramatic improvements 
in performance achieved through a systematic aporoach to course de- 
sign which concentrates on the learner, not the teacher, which spec- 
ifies revisions in the materials and approach based on data collec- 
ted frem real students. At least tentatively, it seems reasonable 
to adopt the position that one mav safely ignore the nuances per- 
ceive! by many as important in "teaching," and concentrate on de- 

J A much mere comprehensive set of reports and data have been col- 
lected than ceu be reported here. These data have been selected by 
the author for the eurpese of illustrating specific features of 
course design. The conclusions drav;n are those of the author and do 
not necessaril> reflect the official position of the U. c. Naval Acad- 
emy, t;,e Now Ycrk Institute of Technology, or the U. S. Office of Ed- 
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signing experiences intended to produce specific performances in 
learners*. 

This research was concerned with one aspect of a total effort 
to design a multi-media physics course, that of collecting data 
specifically on the learning materials used in the Fall of 1969. 
Further, it was planned that preliminary comparisons would be made 
between the traditional physics course and the multi-media course 
under development.. 

The 1969 tryout followed two earlier tryouts of similar mater- 
ials on a much smaller scale* These earlier tryouts were concerned 
with the level of content and the methodological problems of imple- 
menting the multi-media coure The Fall 1969 tryout was designed 
to collect data on the learning materials and procedures, and the 
Spring 1970 tryout provided the first opportunity to conduct the 
course on a relatively self-paced basis# 

The original and subsequent versions of the course weu*e design- 
ed according to the procedures set forth in the Empirical Course 
Development Model (Deterlire ard Branson, l>7l) , a specific reouire- 
ment of which is the continuous recycling of the course to insure 
that planned improvements are made during each iteration. The model 
provides for the repeated collection and analysis of three distinct 
types of data; time, performance, and rating. 

Time data are collected principally when the subject is actually 
in contact with the unique feature of a particular experimental con- 
dition. Performance data are collected regularly cn as saturated 
a basis as time permits. Rating data are collected on student con- 
fidence, preference, and estimates of difficulty. Repeated analyses 
of this data and continuous revision of the course ultimately should 
tend to optimize performance according to the requirements found 

2 



3 



in subsequent years. 

The Fall 1969 tryout ^as conducted under actual operating con- 
ditions at the Academy, except for the rigid scheduling of the stu- 
dents imposed by the experimental design. Ethical considerations 
and Academy administrative requirements made it essential that no 
midshipman be penalized because he was a part of the experiment * 
This rule held true whether the student was a member of the experi- 
mental groups or the control group* Thus, the formative data col- 
lected during this period are of considerably more value than the 
summative data*. 

There are 158 Terminal Objectives {TO f s) of the course, each 
of which 1 s composed of one or more enabling objectives which may 
or nay not be hierarchical. The TO's were ordered according to a 
strategy developed by Finkel (1969) and learning materials were 
developed to produce the desired level of student performance on 
these objectives.. 

This report is concerned vlth the specific data collected on 
each of t..e learning materials, the technical characteristics of 
the cri terl on-referenced test items used as measures of the TC's, 
and the preferences of the students for the alternative approaches 
to study. Incidentally, some interim data of a summative nature 
will be presented to indicate the progress and direction of the 
continuing v;ork# 

The ultimate intention in the project is to provide the Knval 
Academy with an effective and efficient Physics Course tssed on the 
Empirical Development Model, and readily modifiable by the course 
instructors once the final package ha3 been delivered. It is im- 
portant to note that the methods and procedures to be employed in 
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continuously improving the co'irse are thought to be equally im- 
portant to the final package of materials*. 

Since it is planned that the course will be revised regularly, 
any course component should contribute directly, cither independent- 
ly, or in combination, to the achievement of criteria. Other fact- 
ors being equal, those contributing course components which are 
least expensive, or, more easily and inexpensively revised, should 
be selected for the final package. Time, preference, and performance 
measures should all be considered in assembling the final package* 

Hypotheses 

Two bases were used to derive hypotheses: The intended purpose 

of the course, and, the specific characteristics of the parallel me- 
dia used*. 

1. The experimental groups will have a higher population mean 
performance on the Final Examination. 

2. The experimental groups will have a smaller population var- 
iance than will the control groups on the Final Examination. 

3. Audiovisual groups will have a higher population mean sccre 
on motion dependent posttest items than will ether parrallel media 
groups pooled. 

4. Tnere will be no population mean difference between parall- 
el media groups pooled and other experimental groups pooled on diff- 
icult media-relate 1 posttest items. 

5. There will be no population mean difference between the 
parallel media groups pooled and the other experimental groups pool- 
ed on all media-related posttest items,. 
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Method 



SUBJECTS 

The subjects were 371 second year midshipmen at the U.B*. Naval 
Academy enrolled in the S211 Physics course during the Fall of 1969.. 
This number includes students who took the Final Examination and on 
whom background data was available*. 

M ATERIALS 

The learning materials included four widely used basic Physics 
textbooks selected by the Faculty of the U# S* Naval Academ; for the 
S211 course.. These textbooks served as the source of content from 
which other learning materials were developed* 

Content in the textbooks was selected and converted to Terminal 
Objectives (TO f s),. These TO's were then -organ! zed and seouencud log- 
ically. A Study Guide was developed from the objectives, containing 
problems and solutions, and additional elaboration of the content, 
and was programmed with branched remedial? according to a scrambled 
book format. 




The content of the Study Guide was then analyzed to select topics 
for additional elaboration in the "parallel media.' 1 The "parallel 
media" consisted of: a) Videotape recordings of 15-40 minute duration, 
b) Talking Books, consisting of still visual adaptations of the video 
portion of the videotape, with an auditory commentary scripted from 
the audio portion of the videotape, and c) Illustrated Eo-cks, in which 
visuals from the Talking Book were emphasized and highlighted, end, 
the audio ~ortion was condensed end printed with graphic emphasis. 

Four criteria were used to make this selection of content ( mot ion 
dependent; difficulty; Academy experience; course balance.) This 
selection procedure is elaborated on in Doterline and Branrcn (1969). 

In addition to the "canned" materials, two typers of lecture were 
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also used. First, a series of lectures with specific demonstrations 
were prepared and based on the TO 1 s for a given lesson. Then, a 
series of lectures, without demonstration, was also prepared on the 
same TO*s. 

RESPONSE MEASURES 

Student performance in the course was measured by 159 criterion 
referenced constructed response items administered ten items per week 
except for weeks A £, and H, where 12, 7, and 20 items respectively 
were given. Students were asked to make three entries on the answer 
sheet provided for each question: their answer to the question, their 
subjective confidence according to a method derived from Shuford and 
Wassengill, and finally by their rating of the item difficulty on a 
five point scale<. Thirty of these 159 items tested TC*s on which 
parallel media p 'esentations were available. 

Following the completion of the course, the students were given 
a 60 item Final Examination composed of half multiple-choice and half 
const ructed-respon3e items, *he Final Examination w'as used princi- 
pally to determine the student 1 s grade in the course. 

The time students spent in contact with the experimental mater- 
ials or procedures was recorded by proctors, Each Monday, during the 
second half of the course, students were asked to complete a 13 item 
rating scale designed to obtain student reactions to specific feature 
of the experimental conditions. 

Each test question was rated by the Academy faculty prior to its 
administration on two five point scales: relative difficulty in Math- 
ematics, and relative difficulty in Physics. Each ite~* was also exam 
ined lor being inappropriately easy or difficult and for being irrel- 
evant to the TO being tested. An average difficulty rating of Physic 
O and Mathematics was calculated for each test item. Pophara and Husek 
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(1969) have discussed this approach more completely. 

The programmed Study Guides used n wet-to-reveal" answer sheets* 
When the student answered a question, he marked a chemically treat- 
ed box,. If his answer was correct, he was directed to the page con- 
taining the next question, If wrong, he was directed to a remedial.. 
Y/hen he completed the remedial, he then answered again on the chem- 
ical answer sheet. This procedure was repeated until he was correct, 
or made a total of four responses. The answer sheets were optically 
scanned and the number of answers actually revealed were counted for 
each of the 600 plus Study Guide problems. 

In addition to the performance, preference, and time data, other 
measures were available on the midshipmen; 

SAT Verbal and Mathematics ; Strong Vocational Interest Blank; Qual- 
ity Point Ratio; Whole Man Score; Physics Department Validating Exam- 
ination; High School Rank; Academy Naval Scale,. It was felt that in- 
dividual differences might play some part in performance or prefer- 
ence of the various experimental conditions. The rationale for the 
selection of the measures and the possible Implications of their use 
is detailed in Dcterline and Branson (196?), ar.d Brar.son and Deter- 
line (1971). 

PROCEDURE 

Cne-hundred eighty one midshipmen were r&ndcmly assigned to seven 
groups, There were seven experimental conditions allowing each group 
a different treatment each week for seven weeks. The 16 week Acad- 
emy semester was divided into two seven-week blocks, allowing for a 
week of review after each seven week block, The groups were assign- 
ed to conditions by the selection of two Latin Squares according to 
the procedure described in Deterline and Branson (1969). The Latin 
^ Squares were used to balance order ar.d sequence effects of present- 




and to permit compliance with Acsde-y policy of making all 
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materials available to all midshipmen.. 

Each Saturday, the posttest for that week was given immediately 
following the administration of the pretest for the coming week. Dur- 
ing the semester, the combined testing time for pre and posttests could 
not exceed one hour,. 

Each experimental group was identified by the feature distinguish- 
ing it as unique: Audiovisual (AV) received the videotapes; Talking 

Book { TB) ; Illustrated Book (IB); Lecture Demonstration (LD) ; Study 
Guide ( SG) ; Lecture { L) ; Student Option (SO) was free to use any or 
none of the prescribed materials. 

Posttests were scored immediately upon completion by the students, 
and scores were posted by 8:C0 A. M. Monday. In addition to the mid- 
shipman fe score, a listing of the TO's on which he missed questions 
was printed on the same sheet. If a student missed more than three 
questions, he was requested to make an appointment with his instruc- 
tor for a remedial session. If he missed more than five, he was re- 
quired to attend such a session,. 

One-v;ay analyses of variance were performed on both the confid- 
ence modified scores and the proportions correct using, from the Latin 
Square presentation sequence, groups, weeks, and conditions as treat- 
ments. These analyses were performed independently on the media- re- 
lated items and the total pcsttest scores. Media related items are 
a subset of the total test score,. 



HYPOTHESIS TESTS 

The null form of Hypothesis 1 was not rejected on the basis of 



The null fern of Hypothesis 2 was r:Jected on the basis of a var- 



q \ance ratio of 1.56. which with 1^5 and l£c df, is significant at ,01 



Results 



an inadequate t value, t^l.O 




8 



9 

The null form of Hypothec! 3 3 was not testable since the 3ample 
mean difference was the opposite of the hypothesized direction. 

No decision on Hypothesis 4 was made since t, < 1.0.. 

Hypothesis 5 was rejected (using a two-tailed test) on the basis 
of a mean difference favoring the non-parallel media conditions (J. = 
-28.36, 76 df , p <«005).- 

Since the variances were different and the mean3 were not, a poss- 
ible contributor to that increased variance might he different perfor- 
mance on the two sub-tests contained 5n the Final Examination. If the 
distribution of Final Examination scores was bimodal for the control 
students and not for the experimental students, one Possible explana- 
tion of the increased variance mi,$ht be offered.. The control group did 
significantly better on raultlple-ohoice questions than they did on con- 
structed-response questions (t = 4.15, 3£2 df, p<*005). This differ- 
ence did not hold for the experimental groups (t = 1 . 52 , 150 df, p> 
.05).* 

The one-way analyses of variance were concerned with major group 
differences on the total scores. First, the F for experimental con- 
ditions was not significant, neither for proportion correct nor for 
confidence modified scores, in both cases belngO.O., 

The F’s for weeks were both significant beyond the .01 level, 
probably indicating, since all treatments occurred in all weeks, a 
difference in difficulty level of the Physics 1 materials (proportion 
correct, F = 331. 5» <3f 5, £45 , p <.C1; confidence, F = 34.6, df 5^ 545, 
P <.01). 

T ERMINAL CEJECriVES 

For each of the TO*?, as represented by the posttest ite-se end 
the Study Guide responses, the following data were collected for re- 

O 
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vision purposes; 



proportion correct 
confidence nodified scores 
difficulty rating in Physics 
difficulty rating in Mathematics 
student difficulty rating 
recorded confidence 



Table 1 indicates the correlation matrix among these variables* 
Faculty Physics and Mathematics ratings show moderately high (.66) 
linear relationship* Also of note is the r of .66 between the stu- 
dent's confidence and his rating of difficulty. He was more confident 
of the difficult items*. It also appears that the faculty was a better 
Judge of the difficulty of the items than was the student*. All of 
these relationships are high enough to be useful in course revision, 
BACK GROUND VAR I ABLE 5 

In searching for data upon which to base media decisions, it was 
thought that an analysis of the relationships among background and 
performance variables would improve predictions* Table 2 indicates 
the intercorrelations among 9 selected variables* While none of -the 
correlations is surprisingly high or low. 



the lack of relationship between the Final ExaTinaticn ar.d the total 



course designers*. 

FRgPPgSNCS and TIMS DATA 

Itor.s on toe rating scale were combined for each experimental 



Insert Table 1 about nere 



Insert Table 2 about here 
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group and a single dimension of favorable-neutral-unfavorable was 
used for between group comparisons* Each group could be described 
with three scores: proportion favorable, proportion neutral, and pro- 
portion unfavorable. Two rankings were made, the first ranked the 
groups on the favorable proportion, the second ranked the groups on 
the unfavorable proportion. 



Insert Table 3 about here 



Table 3 reveals the Lecture and the Student Option conditions 
to be essentially tied and rated both as most favorable and least un- 
favorable. The L/SGr condition earned the opposite rankings: least fa- 
vorable and most unfavorable*. 

While the favorable and unfavorable are two ends of the same con- 
tinual, and are thus not independent, both rankings were made to elim- 
inate the influence of neutral responses# 

Total time spent in each experimental condition by all students 
was computed and rank-ordered from least to most. Keen time per con- 
dition was 171 minutes with e standard deviation of 61 minutes. Lec- 
ture and Student Option conditions required the least amount of time, 
while the L/sG condition required the most. The rank correlation (rho) 
between preference and time was .C7, allowing rejection of the null 
hypothesis (Ho: rho=0) at the .02 level.. 

Discussion 
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It was not the purpose of the tryout to arrange experimental con- 
ditions v/hich would produce statistically significant differences* 
Rather, the purpose was to gather data which would be useful In re- 
vising the course to make It mors appropriate for student-paced use# 
Procedural ly , the kind? of data collected must te relatively inexpen- 
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sive and require minimum time. Successive iterations of the course 
are not likely to improve student performance if such data cannot be 
used for purposes of revision* 

It is one purpose of the course to increase mean student perfor- 
mance and reduce the variation in group performance* To that end, the 
rejection of Hypothesis 4 indicates that progress v/as made*, While the 
difference between means favored the experimental groups, the differ- 
ence was not statistically significant. 

Not unexpectedly, all experimental groups did as well on those 
Final Examination questions requiring constructed response answers as 
they did on the multiple-choice questions*. This was not true of the 
control groups, even though no correction for guessing was made. 

The experimental conditions were all apparently equally effec- 
tive in teaching students the required criterion behavior. It should 
be noted, however, that the criteria were based on a highly limited 
range of responses: the working of Physics problems. This conclusion 
seems warranted, regardless of whether one uses the norm-referenced 
Final Examination, or the ciiterion referenced total posttest scores, 
In the special case of the media related test items, the non-audiovisua 
groups did significantly better In total performance. 

If one considers the performance data in light of the preference 
data, it appears that students are concerned with tnose experiment al 
conditions which take the least time and vrhich are most directly re- 
lated to the content of the tests. For example, the L/SG ccr.ditlon 
we * considerably less attractive to the students than was t:e straight 
Lecture (L) group, Conceivably, while the de^enstrati on may have been 
interesting, the students viewed it ar having no relationship to the 
Important criteria of the course, namely, the working of Physics pro- 
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blems. While the inadequacy of such criteria has been discussed more 
fully elsewhere (Branson, 1970) they are, nevertheless, widely used. 

The preference of the SO condition may be attributable to the 
small amount of time actually prescribed for the students during those 
weeks. That is, if the lecturer is willing to show students how to 
work problems, he is willing to listen. However, if one burdens the 
student with demonstrations, Audiovisual presentations, etc,, the stu- 
dent seems much more willing to do it himself#. 

Regardless of the intergroup comparisons, the data collected are 
quite interesting* Each Terminal Objective was treated in a variety of 
ways* in the Study Guides, textbooks, and the lectures. The criterion 
referenced test items used to measure the behavior were evaluated by 
the faculty along a number of dimensions: appropriateness to the TO 
(content validity), difficulty in Mathematics, difficulty in Physics. 

These ratings are extremely valuable in providing a methodology 
by which a faculty member can, a priori, determine the level at which 
his course is taught. Provide \ that one is veiling to acrept final 
performance cf the students as an indication of the levsl of sophisti- 
cation of the course, the degree tc which this can be predicted in ad- 
vance is a good indicator of the course “level. " 

If, on the other hand, it is necessary to wait until after the re- 
sults are in to specify the level, it appears that the students, not 
the faculty, decide what level of performance is acceptable. Parti- 
cularly, if the grades In the course are assigned on any "normal” curve 
tasi s. 

Our results indicate that the faculty is considerably tetter at 
predicting student performance on the basis of difficulty ratings than 
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the students are. Faculty correlations were -.42 and -.58 between 
performance and difficulty, while students difficulty and performance 
correlated only -.25. 

Tills procedure for establishing course difficulty level appears 
emxinently more desirable than a method which uses ad hoc student per- 
formance, to determine which test items should be retained and discard- 
ed*. Our results indicated that there was a significant "weeks" effect, 
from which we inferred that weeks were not equally difficult. Physi- 
cists confronted with this data claimed to have known all along that 
some topics were indeed more difficult than others, as is virtually 
always the case in academic subjects. 

The fact that they could predict, with reasonable precision, the 
level of difficulty of the test items, and, .thus, control this level 
of difficulty, transfers the responsibility of course level det- ruin- 
ation to the faculty.* 

The Study Guide results were of great general interest. While 
the "Linear-Branching" programmed instruction controversy has been 
deed for many years, it appeared reasonable in this course to offer 
specific remedial frames, to which the student was looped, when he 
failed to answer :orreetly on the first attempt. Further, that more 
specific remedials would be more effective than general remedials. 

While the data for each Volume of the Study Guide has been presented 
elsewhere, an analysis of Volume N is interesting at this point. 

Volume N had "general" remedials. That is, the remedial was simply 
a presentation of the correct way to work the problem. The remainder 
of the course used specific remedials. That is, each problem was anal- 
ysed and the most likely, common, and protable errors ver. selected 
for elaboration. The students were shown why they were wrong, not 
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how to do the problem correctly*. 

If a remedial is effective, it ought to reduce the probability 
of error on the subsequent attempts at the answer. Thus, if a stu- 
dent has missed the correct answer on the first trial and is given a 
remedial, he ought to have a better chance to be right on the second 
attempt than someone not receiving the specific remedial* We have 
used the following formula to assess the effectiveness of remedials, 
where 

number of double choices 

Effectiveness = — ■ — 

total number of double, triple, and Quadruple 
choices 

The results for Volume N indicated an effectiveness index of 
♦ 59 and the general results of the course indicated an effectiveness 
index for the remaining Volumes of .60. On the basis of this data, 
it was decided not to include specific remedials dealing with student 
errors in subsequent versions of the course. Course developers would 
concentrate on a more careful description of the correct way of work- 
ing the problems,* 

Finally, the very low correlation between the performance of stu 
dents on the total of 159 criterion referenced items and the 60 item 
norm-referenced Final is encouraging. . Frofessors judgment of perfor- 
mance on criterion referenced items is a better indicator of final 
score on these items than is total student performance cn norm-refer- 
enced items used as a predicator,. Since the posttest items had been 
carefully screened for content validity prior to their inclusion on 
the test, and had been Judged according to their expected level of 
difficulty, it was possible to make a more accurate determination of 
the actual course level of difficulty than would otherwise have been 
possible. 
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Subsequent versions of the course can use the test items in a 
pretest form and establish a baseline of student performance, having 
available past performance on the same items as a comparison. It is 
important to note here that professor Judgment, tempered by past ex- 
perience, is the critical element in developing the criterion measures. 
Student performance alone is not used* Consequently, test items are 
not discarded when a large number or proportion of students answers 
them correctly. They are discarded when thev are rated and Judged in- 
appropriate by the faculty. 

The results of the Fall 1969 tryout demonstrated to the Physics' 
faculty that the method of instruction was not the critical element in 
student performance, an accomplishment of some magnitude. Further, 
that students could, when provided with the he cesshry instruction 
and materials, achieve good results on their own., And finally, that 
if data is collected systematically and used to revise the course com- 
ponents, improvements can be made at each successive iteration., 
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Table 1 



Correlation Matrix for the Measures 
Taken on Each of the Posttest 
1 terns 



Variable 


X 1 




3 


4 1 


5 * 


Performance* 
Mean Log 
Confidence 


Math 

Difficulty 


.Physics 
; Difficulty 1 


j Mean 

Confidence 


Student 

Difficulty 


i 




-.42** 


-.56** 


,03 


-.25** 


2 






.66*# 


-.11 


.10 


3 




i 




. .07 


. 24** 


4 


J 








. 66** 


5 













Note.f- 

** P < .01 

The negative correlations Indicate thst as difficulty 
ratings Increase, performance decreases,. 
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Table 2 



Lntercorrelations of Background and Performation Variables 
on Those Subjects From Whom a Complete Set of Data 
Was Available N = 77 



- — — — ■ 


: r —i— - ■ — — -< 


1 


2 


3 


4 


5 


6 


7 


, _ - - 


... . 







I 




.... . 







1. 


SAT Verbal 
















2. 


SAT Math 


.31 














3. 


Highschool Rank 


.14 


.06 












4. 


Whole Man 


.17 


.32 


.57 


j 






i 


5. 


Quality Point Ratio 


.22 


.28 


.43 | 

i i 


j • 53 j 


i 

i 




! 

i 


6. 


Final Exam 


.34 


.39 


.25 | 


• 27« j 


.70 






7. 


Physics Validation i 


! r ^ 


.36 


.20 ' 


i ! 

.23. i 

l 


.38 i 

j 


: .52 


1 

l 

1 


6. 


Media Related 

| 


.05 

t 


| 


f .03 
| 


r> r\ f 

|4C j 

i 


.28 | 


.23 1 


.04 j 

{ 


9. 


j 

Total Posttest 


O 

• 

i 


.06 


|.09 j 


| . 35 j 


.40 j 


r* r i 

• C 5 : 

1 


: .11 ! 

; i 




i 


i . 




t .J 


1 .. . 1 .. 1 


j ] 


\ .. t 



Note.- For 70 df, the .05 level is .23, t'r.e .01 level Is .30. 



O 
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Table 3 



Preferences and Tlse Data and the Rank Orderings 



of Their Proportions 





Proportions 


Rank 


Order 


Time 




Favorable 


Neutral 


Unfavorable 


Most 

Favorable 


Least 

Unfavorable 


(Least 
To Most) 


1 

Lecture i 


i . 50 * 


. 32 * 


.15* 


1 


1 


2 


i 

Student i 
Option | 


! 

1 

| .47* 


. 34 * 


.19* 


2 


2 


1 


Talking 
Book j 


1 

| 


.35 


.23 


3 


3 


: 4 


Study 

Guide 


1 j 

\ .41 


i 

: .35 


.24 ; 

j 


* j 

4 I 

! 


f 

4 


3 


Audio- 

Visual 


i 

! . 34 


I 

; .41 


j 

1 

• 25 ! 

[ \ 


i 

5 j 


1 

1 

i 

5 j 


6 


Illus- 

trated 

Book 


.33 


: 

1 

1 

: .40 


I i 

1 

1 .27 j 


i 

6 i 


t 

i 

6 ; 


5 


Lecture/ 

De-on- 

stratlon 


. c6* 


i 

. 40* 


1 * 

: . 34* 


? 1 


7 j 


7 


Mean 


. 39 


' .37 ! 

■ 


1 .24 | 








Standard ■ 
Deviation ,08 


; i 


i > 

i .05 









. i 



Note.- 

*■ Indicates a deviation of£l S. D. 
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